| | --- |
| | license: llama3.1 |
| | datasets: |
| | - RUCKBReasoning/TableLLM-SFT |
| | language: |
| | - en |
| | base_model: |
| | - meta-llama/Llama-3.1-8B-Instruct |
| | tags: |
| | - table |
| | - QA |
| | - Code |
| | --- |
| | |
| | # TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios |
| |
|
| | | **[Paper](https://arxiv.org/abs/2403.19318)** | **[Training set](https://huggingface.co/datasets/RUCKBReasoning/TableLLM-SFT)** | **[Github](https://github.com/RUCKBReasoning/TableLLM)** | **[Homepage](https://tablellm.github.io/)** | |
| |
|
| | We present **TableLLM**, a powerful large language model designed to handle tabular data manipulation tasks efficiently, whether they are embedded in spreadsheets or documents, meeting the demands of real office scenarios. TableLLM is fine-tuned based on [Llama3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct). |
| |
|
| | TableLLM generates either a code solution or a direct text answer to handle tabular data manipulation tasks based on different scenarios. Code generation is used for handling spreadsheet-embedded tabular data, which often involves the insert, delete, update, query, merge, and plot operations of tables. Text generation is used for handling document-embedded tabular data, which often involves the query operation of short tables. |
| |
|
| | ## Evaluation Results |
| | We evaluate the code solution generation ability of TableLLM on three benchmarks: WikiSQL, Spider and Self-created table operation benchmark. The text answer generation ability is tested on four benchmarks: WikiTableQuestion (WikiTQ), TAT-QA and FeTaQA. The evaluation result is shown below: |
| |
|
| | | Model | WikiTQ | TAT-QA | FeTaQA | WikiSQL | Spider | Self-created | Average | |
| | | :------------------- | :----: | :----: | :----: | :-----: | :----: | :----------: | :-----: | |
| | | TaPEX | 38.6 | – | – | 83.9 | 15.0 | / | 45.8 | |
| | | TaPas | 31.6 | – | – | 74.2 | 23.1 | / | 43.0 | |
| | | TableLlama | 24.0 | 22.3 | 20.5 | 43.7 | - | / | 23.4 | |
| | | TableGPT2(7B) | 77.3 | 88.1 | 75.6 | 63.0 | 77.34 | 74.42 | 76.0 |
| | | Llama3.1 (8B) | 71.9 | 74.3 | 83.4 | 40.6 | 18.8 | 43.2 | 55.3 | |
| | | GPT3.5 | 58.5 | 72.1 | 71.2 | 81.7 | 67.4 | 77.1 | 69.8 | |
| | | GPT4o |**91.5**|**91.5**|**94.4**|<ins>84.0</ins>| 69.5 |<ins>77.8</ins>|<ins>84.8</ins>| |
| | | CodeLlama (13B) | 43.4 | 47.3 | 57.2 | 38.3 | 21.9 | 47.6 | 43.6 | |
| | | Deepseek-Coder (33B) | 6.5 | 11.0 | 7.1 | 72.5 | 58.4 | 73.9 | 33.8 | |
| | | StructGPT (GPT3.5) | 52.5 | 27.5 | 11.8 | 67.8 |**84.8**| / | 43.1 | |
| | | Binder (GPT3.5) | 61.6 | 12.8 | 6.9 | 78.6 | 52.6 | / | 36.3 | |
| | | DATER (GPT3.5) | 53.4 | 28.5 | 18.3 | 58.2 | 26.5 | / | 33.0 | |
| | | TableLLM-8B (Ours) |<ins>89.1</ins>|<ins>89.5</ins>|<ins>93.4</ins>|**89.6**|<ins>81.1</ins>|<ins>77.8</ins>|**86.7**| |
| |
|
| | ## Prompt Template |
| | The prompts we used for generating code solutions and text answers are introduced below. |
| |
|
| | ### Code Solution |
| | The prompt template for the insert, delete, update, query, and plot operations on a single table. |
| | ``` |
| | [INST]Below are the first few lines of a CSV file. You need to write a Python program to solve the provided question. |
| | |
| | Header and first few lines of CSV file: |
| | {csv_data} |
| | |
| | Question: {question}[/INST] |
| | ``` |
| |
|
| | The prompt template for the merge operation on two tables. |
| | ``` |
| | [INST]Below are the first few lines two CSV file. You need to write a Python program to solve the provided question. |
| | |
| | Header and first few lines of CSV file 1: |
| | {csv_data1} |
| | |
| | Header and first few lines of CSV file 2: |
| | {csv_data2} |
| | |
| | Question: {question}[/INST] |
| | ``` |
| |
|
| | The csv_data field is filled with the first few lines of your provided table file. Below is an example: |
| | ``` |
| | Sex,Length,Diameter,Height,Whole weight,Shucked weight,Viscera weight,Shell weight,Rings |
| | M,0.455,0.365,0.095,0.514,0.2245,0.101,0.15,15 |
| | M,0.35,0.265,0.09,0.2255,0.0995,0.0485,0.07,7 |
| | F,0.53,0.42,0.135,0.677,0.2565,0.1415,0.21,9 |
| | M,0.44,0.365,0.125,0.516,0.2155,0.114,0.155,10 |
| | I,0.33,0.255,0.08,0.205,0.0895,0.0395,0.055,7 |
| | ``` |
| | |
| | ### Text Answer |
| | The prompt template for direct text answer generation on short tables. |
| | ```` |
| | [INST]Offer a thorough and accurate solution that directly addresses the Question outlined in the [Question]. |
| | ### [Table Text] |
| | {table_descriptions} |
| |
|
| | ### [Table] |
| | ``` |
| | {table_in_csv} |
| | ``` |
| |
|
| | ### [Question] |
| | {question} |
| |
|
| | ### [Solution][INST/] |
| | ```` |
| | |
| | For more details about how to use TableLLM, please refer to our GitHub page: <https://github.com/TableLLM/TableLLM> |
