Can Codex Handle Real-World Data Analysis?

Authors: Zhisong Qiu, Jiafu Wei, and Chen Yan
DOI: 10.5281/zenodo.21262548

Data analysis is an essential task in the real world. Whether performed by professional data scientists or engineers in vertical domains such as finance and healthcare, it typically involves collecting, cleaning, organizing, modeling, and interpreting data to uncover patterns, validate hypotheses, and support decision-making.

However, completing an end-to-end data analysis task is far from trivial. A seemingly simple question may involve multiple steps, including data source selection, schema understanding, statistical modeling, and result interpretation. If any one of these steps is misunderstood, the final conclusion may deviate from the user’s actual intent.

As large language models become more capable, end-to-end data analysis agents are becoming increasingly feasible. Ideally, users should be able to describe a question in natural language, and the agent should automatically find data, write code, debug scripts, generate charts, and produce a report.

Recently, Codex introduced a new data analysis plugin. We therefore conducted a set of tests around Codex with the data analysis plugin enabled, aiming to answer a direct question:

Can frontier models such as Codex reliably complete real-world data analysis tasks today?

Our tests cover two types of scenarios.

• The first category focuses on business and everyday data analysis tasks, including complex business rules, heterogeneous files, open-ended analytical reports, and multi-table joins.
• The second category focuses on scientific and biomedical data analysis tasks, including real biological data analysis and medical calculations.

We organize the discussion around the following questions:

• What data analysis capabilities did we test?
• What kinds of data analysis tasks is Codex good at?
• Where do Codex’s most common errors come from?
• What can we learn from a simple prompt comparison experiment?
• How can we use Codex more effectively for data analysis?

Our conclusion is:

Codex has reached a practically useful level for data analysis, especially on tasks with clear rules, visible data, and well-defined schemas. However, to make the results reliable, humans still need to provide metric definitions, domain context, data selection criteria, and validation procedures.

---

What We Tested

This evaluation covers two scenarios: real-world business data analysis and real-world scientific data analysis. The former is closer to self-service analytics in enterprise settings, while the latter is closer to scientific analysis workflows performed by researchers. We sampled tasks from multiple benchmarks.

A few clarifications are needed:

• MedCalcBench tests medical calculation after the relevant entities are already given. It does not test entity extraction from full clinical notes.
• MedAgentGym and OpenProblems are supplementary analyses. For MedAgentGym, Codex was used to analyze failure modes in public trajectories. For OpenProblems, Codex was used to parse leaderboards and summarize them into reports.
• In DSBio, 13 questions could not be solved because external data sources were missing or the required information could not be found. We therefore report results over the remaining 77 questions.
• Tasks from DABstep, DSBench-DA, DAComp-DA, and CoDA-Bench were manually sampled. The selection criteria were real-world relevance and non-trivial analytical difficulty.

---

What Data Analysis Tasks Is Codex Good At?

Overall, Codex performs well on tasks where the rules are clear, the data is visible, and the schema is explicit.

First, Codex is strong at executing clearly specified formulas and rules. On MedCalcBench, Codex achieved an overall accuracy of 83.7%. It solved 60/60 date-related tasks and 239/240 physical measurement tasks. For common medical formulas such as BMI and Creatinine Clearance, Codex usually completed the calculations correctly. We saw a similar pattern in DABstep: when fee rules, routing conditions, and aggregation logic were explicitly stated in the prompt, Codex could parse the rules and complete the computation.

Second, Codex is stable at reading heterogeneous files and extracting structured information. It solved all 8 DSBench-DA tasks correctly. These tasks involve Excel files, image-based rules, and complex table structures. Codex was able to write scripts, enumerate candidate outputs, validate intermediate results, and extract structured information from complex files.

Third, Codex performs well on conventional statistical analysis when the schema is clear. In BioDSBench, the 27959731 task group achieved a 6/10 pass rate, covering disease distribution, TP53 mutation patient counting, response group visualization, Tukey HSD, and related analyses. These tasks are close to standard pandas / statsmodels workflows. In DSBio, the Gut Tissues task group achieved 9/10. This task involved statistics such as Kruskal-Wallis, Chi-Square, Jaccard, Pearson, and Donor Variance Ratio. When the problem statement clearly specifies the analysis target, grouping strategy, and statistical metric, Codex performs relatively well.

Fourth, Codex can conduct open-ended exploration and generate analytical reports. Across the 5 DAComp-DA tasks, Codex was able to read SQLite databases, inspect schemas, define metrics, run analyses, and generate well-structured reports. This suggests that Codex already has meaningful end-to-end analytical capability, including data exploration, metric computation, chart generation, and conclusion writing.

Fifth, Codex is good at generating code scaffolds and local functions. In BioCoder, Codex passed syntax and interface checks for 207/207 tasks. This indicates that it can understand function signatures, input-output formats, and basic implementation structures. However, only 20/207 outputs matched the reference implementations. This shows that a correct code scaffold does not imply fully correct functionality. For tasks that require exact bioinformatics algorithm details, edge-case handling, and strict data format alignment, further validation is still necessary.

Finally, Codex is strong at parsing structured trajectories and leaderboards. The supplementary analyses on MedAgentGym and OpenProblems show that when information is already organized into structured trajectories, tables, or leaderboards, Codex can reliably read, compare, summarize, and explain it.

---

Where Do Codex’s Most Common Errors Come From?

We analyzed errors across both business and scientific analysis tasks. The major failure modes can be grouped into four categories.

1. Concepts and Entities Are Not Aligned

Real-world business questions are often expressed in natural language, while database concepts are rarely so direct. For example, when a user asks “How many active users do we have?”, does “user” mean login account, device, or paying customer? Should fraudulent accounts, internal test accounts, and refunded customers be excluded? If the user does not specify these details, Codex may misunderstand the intent and use the wrong data.

CoDA-Bench 321 is a representative case. The question asks: “What percentage of single-category orders received a 5-star rating?” Codex defaulted to filtering out orders without reviews, and computed the 5-star ratio only among reviewed orders. However, the intended interpretation was that orders without reviews should remain in the denominator. DAComp 004 shows a similar issue. The task asks for an analysis of repurchase rate and correlation for “monthly champion products.” Codex expanded the correlation analysis from monthly champion products to all products, which led the analysis away from the intended scope.

2. Multi-Table Joins Go Wrong

Multi-table joins are the most common source of silent errors in business analysis. For example, CoDA-Bench 327 asks for the average freight cost per order. The correct approach is not to simply average by the number of orders; instead, you should use the order table as the primary table, join it with the order detail table, and then use the number of rows after the join as the denominator. Codex might interpret the denominator as the count of distinct orders, or it might keep only those orders that have line items after the join. While the SQL won't throw an error, the result is already wrong.

This issue is common in real companies. Order tables, order item tables, payment tables, refund tables, and logistics tables often have one-to-many, many-to-many, or delayed reconciliation relationships. When a data analysis agent handles such tasks, humans need to reduce silent errors by specifying the intended granularity upfront and validating the intermediate results afterward.

3. Implicit Formulas and Metric Definitions Are Underspecified

In many tasks, failures do not occur because Codex cannot perform data analysis. Instead, they occur because the metrics, terms, or formulas in the task can be defined in multiple ways. If the user does not explicitly define them, Codex will choose a definition that appears reasonable, but that definition may not match the user’s expectation.

MedCalcBench failures are concentrated in tasks that require external formula tables or segmented scoring rules. Delta Gap and Albumin Corrected Delta Gap are often not simple algebraic formulas. They involve details such as segmented scoring, threshold accumulation, unit conversion, normal-value baselines, and sign conventions. If Codex relies only on the task statement and its prior knowledge, it can easily compute the wrong result.

DSBio 33 reflects a similar issue. The task asks about fold-change enhancement in protein-coding genes, but it does not explicitly define “enhancement.” Codex interpreted it as “how much extra enhancement occurs when IFNg is combined with infection compared with the two individual conditions,” and implemented it as (IFNg_SL1344 - IFNg) - (SL1344 - naive) The reference answer, however, used a different metric definition: directly comparing SL1344 and IFNg_SL1344 under the infection condition.

4. The Data Source Itself Can Be Wrong

Real scientific datasets often contain many versions of data: raw counts, processed matrices, sample metadata, and different experimental objects. The input data provided by the user may contain noisy or misleading files, and the model cannot derive the correct answer from the wrong data source.

For example, in DSBio 34, Codex followed the CQN matrix mentioned in the task and obtained all-zero results. Only after switching to the count matrix did it obtain the reference-consistent IFNg_SL1344 result. The 13 unsolved DSBio tasks are similar: in some cases, the problem was not the model’s analytical capability, but the fact that the data source did not contain the required labels, comparison relationships, or raw files.

This shows that data source quality has a major impact on agent performance. Clean, accurate data sources with the necessary context can substantially improve the success rate of data analysis agents.

---

A Prompt Comparison Experiment

To further understand the sources of failure, we ran a simple comparison experiment on business analysis tasks. We manually added the correct metric definitions to the prompt, then asked Codex to rerun previously failed tasks.

• coda-321, dacomp-008, DABstep-2768, and DABstep-1457 were successfully solved.
• dacomp-004 partially improved. The analysis scope became clearer, though there was still room for improvement.
• coda-327 still did not fully succeed, indicating that multi-table joins and granularity understanding remain difficult.

This result suggests that in data analysis scenarios, many failures do not come from insufficient analytical capability. Instead, they come from missing background knowledge such as metric definitions.

However, this problem is extremely common in real-world settings, because users will not write complete prompts every time. Usually, they only provide simple requests such as:

• “Help me look at last month’s sales.”
• “Check how new user retention is doing.”
• “Find recent cost anomalies.”
• “What changed in the markers of this cell population?”

Behind these questions lie many hidden metric definitions and domain assumptions: whether revenue should exclude refunds, whether retention should be measured by calendar week, whether cost anomaly means year-over-year change, month-over-month change, or budget deviation, and which statistical criterion should be used for markers. It is unrealistic to expect users to fully specify all of these details in every prompt.

---

How to Use Codex More Effectively for Data Analysis

Based on the test results, we summarize three practical recommendations.

1. Provide Accurate Data Source Information

We recommend that the first step of data analysis should be to provide accurate data source information. Many errors arise because Codex cannot derive the correct answer from incorrect or inaccurate data, or because it misunderstands the data definition. The goal of this step is to reduce Codex’s need to guess what the data means.

Codex can infer data structures from file names, column names, and sample values, but this inference may not match the actual business or scientific context. We therefore recommend the following workflow:

1. Ensure that the provided data sources are accurate and clean, and that they contain the data required by the analysis. When possible, provide additional reference data that may be useful. For example, when analyzing repurchase rate, provide not only order details, but also data that identifies users, order time, and repeated purchase behavior.
2. Explain the purpose of each data source. In the prompt, clearly describe what each file or table is used for and how Codex should use it.
3. Define the data and fields. Explain what each row in a database represents, what key fields in a CSV file mean, and how different tables should be joined. This helps avoid misinterpretation of data and fields.

2. Use Skills to Encode Background Knowledge

We recommend using Skills to supplement prompts with background knowledge. Many data analysis errors occur because the user has not provided enough context, so Codex does not know what a business term, scientific formula, or metric definition means in the current setting. A Skill can make this background knowledge explicit in advance and reuse it across similar data analysis tasks.

A Skill should include at least the following types of information:

1. Define terminology. Terms such as “active user” and “repurchase rate” in business analysis, or “fold-change” and “marker enhancement” in scientific analysis, may have multiple interpretations. A Skill should define how these terms are used in the current context.
2. Define metrics. Many common-looking metrics do not have a single universal implementation. For example, should the denominator of repurchase rate include users who purchased only once? Should retention be measured by calendar day or calendar week? A Skill should specify the exact metric definition to prevent Codex from guessing.
3. Provide domain background. This may include domain-specific formulas, segmented scoring rules, experimental designs, or business rules. For these tasks, the necessary background should be written explicitly rather than assuming Codex already knows the complete rule set.
4. Specify reference workflows. A Skill can include a recommended analysis procedure. For business analysis, it may require Codex to first inspect table schemas, then confirm analytical granularity, and only then perform multi-table joins and metric computation. This encourages Codex to follow a fixed process and reduces analytical errors.
5. List common pitfalls. For example, multi-table joins may introduce duplicated counting, and treating order items as orders may change the denominator. If common mistakes are written into the Skill beforehand, Codex is more likely to avoid them during analysis.

The role of a Skill is to make explicit the background knowledge and workflows that users already know but cannot realistically repeat in every prompt. For recurring business or scientific analysis tasks, Skills can reduce errors caused by implicit definitions and make the analysis process easier to audit.

3. Validate the Analysis Process

When using Codex for data analysis, the most important risk to watch for is silent error: the code runs successfully, but some intermediate step, such as data selection, filtering, or analysis logic, is wrong. Therefore, validation should not focus only on the final conclusion. It should also inspect the analysis process.

We recommend checking the following four aspects:

1. Requirement check. Confirm that Codex’s analysis satisfies all user requirements, without omissions or overstated completion.
2. Metric check. Confirm that Codex computes the result according to the intended metric definition.
3. Data check. Confirm that Codex uses the correct files, fields, and filters.
4. Result check. For important conclusions, ask Codex to output intermediate tables, key samples, and calculation traces, so humans can review the full workflow and verify the reliability of the conclusion.

A more realistic positioning is to treat Codex as a data analysis assistant. It is well suited for reading files, exploring data, writing code, and organizing reports. But metric definitions, data source choices, and final conclusions still require human confirmation.

---

Conclusion

Our tests show that Codex already has strong data analysis execution capabilities.

For tasks with clear rules, visible data, and explicit schemas, it can reliably complete file reading, code writing, statistical computation, chart generation, and report organization. For many everyday data analysis tasks, it can significantly lower the barrier to entry, allowing users to complete a relatively full analysis workflow without writing scripts from scratch.

However, the real difficulty of data analysis lies in understanding the metric definitions, data semantics, and domain background behind the question. The same term may have different meanings in different contexts. If these definitions are not explicitly stated, Codex will choose an interpretation that looks reasonable, but that interpretation may not align with the user’s expectation.

Therefore, Codex is currently better understood as a data analysis collaborator rather than a fully automated data analysis expert. It can improve efficiency, reduce repetitive work, and help users quickly explore data. But to obtain reliable conclusions, humans still need to provide clear data source descriptions, explicit metric definitions, and necessary result validation.

A more realistic workflow is a collaboration between humans and Codex:

• Humans define the problem, provide background, confirm metrics, and judge conclusions.
• Codex reads data, executes computations, generates charts, and prepares reports.

In this collaborative mode, Codex already has substantial practical value. It cannot guarantee correctness in every analysis, but when the data sources are clear, the task definition is explicit, and the validation process is sufficient, it can already serve as a reliable data analysis assistant.

If you find this blog useful, please consider citing it as follows:

@misc{qiu2026codex_data_analysis,
  title        = {{Can Codex Handle Real-World Data Analysis?}},
  author       = {Qiu, Zhisong and Wei, Jiafu and Yan, Chen},
  year         = {2026},
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.21262548},
  url          = {https://doi.org/10.5281/zenodo.21262548},
  note         = {Technical blog}
}