is Nikolay Nikitin, PhD. I’m the Analysis Lead on the AI Institute of ITMO College and an open-source fanatic. I usually see lots of my colleagues failing to seek out the time and power to create open repositories for his or her analysis papers and to make sure they’re of correct high quality. On this article, I’ll talk about how we will help remedy this drawback utilizing OSA, an AI software developed by our crew that helps the repository turn into a greater model of itself. For those who’re sustaining or contributing to open supply, this put up will prevent effort and time: you’ll find out how OSA can mechanically enhance your repo by including a correct README, producing documentation, organising CI/CD scripts, and even summarizing the important thing strengths and weaknesses of the venture.
There are a lot of completely different documentation enchancment instruments. Nevertheless, they deal with completely different particular person parts of repository documentation. For instance, the Readme-AI software generates the README file, but it surely doesn’t account for added context, which is vital, for instance, for repositories of scientific articles. One other software, RepoAgent, generates full documentation for the repository code, however not README or CI/CD scripts. In distinction, OSA considers the repository holistically, aiming to make it simpler to grasp and able to run. The software was initially made for our colleagues in analysis, together with biologists and chemists, who usually lack expertise in software program engineering and trendy improvement practices. The primary purpose was to assist them make the repository extra readable and reproducible in a couple of clicks. However OSA can be utilized on any repository, not solely scientific ones.
Why is it wanted?
Scientific open supply faces challenges with the reuse of analysis outcomes. Even when code is shared with scientific papers, it’s not often out there or full. This code is normally troublesome to learn; there isn’t any documentation for it, and generally even a primary README is lacking, because the developer meant to jot down it on the final second however didn’t have time. Libraries and frameworks usually lack primary CI/CD settings equivalent to linters, automated exams, and different high quality checks. Subsequently, it’s not possible to breed the algorithm described within the article. And this can be a large drawback, as a result of if somebody publishes their analysis, they do it with a want to share it with the group
However this drawback isn’t restricted to science solely. Skilled builders additionally usually postpone writing readme and documentation for lengthy durations. And if a venture has dozens of repositories, sustaining and utilizing them might be difficult.
Ideally, every repository needs to be straightforward to run and user-friendly. And sometimes the posted developments usually lack important components equivalent to a transparent README file or correct docstrings, which might be compiled into full documentation utilizing normal instruments like mkdocs.
Primarily based on our expertise and evaluation of the issue, we tried to recommend an answer and implement it because the Open Supply Advisor software – OSA.
What’s the OSA software?
OSA is an open-source Python library that leverages LLM brokers to enhance open-source repositories and make them simpler to reuse.
The software is a bundle that runs by way of a command-line interface (CLI). It may also be deployed domestically utilizing Docker. By specifying an API key in your most well-liked LLM, you possibly can work together with the software by way of the console. It’s also possible to attempt OSA by way of the general public net GUI. There’s quick introduction to most important concepts of repository enchancment with OSA:
How does OSA work?
The Open Supply Advisor (OSA) is a multi-agent software that helps enhance the construction and usefulness of scientific repositories in an automatic manner. It addresses frequent points in analysis tasks by dealing with duties equivalent to producing documentation (README information, code docstrings), creating important information (licenses and necessities), and suggesting sensible enhancements to the repository. Customers merely present a repository hyperlink and may both obtain an mechanically generated Pull Request (PR) with all really useful adjustments or overview the ideas domestically earlier than making use of them.
OSA can be utilized in two methods: by cloning the repository and operating it by way of a command-line interface (CLI), or by way of an online interface. It additionally presents three working modes: primary, computerized, and superior, that are chosen at runtime to suit completely different wants. In primary mode, OSA applies a small set of normal enhancements with no further enter: it generates a report, README, group documentation, and an About part, and provides frequent folders like “exams” and “examples” in the event that they’re lacking. Superior mode provides customers full handbook management over each step. In computerized mode, OSA makes use of an LLM to research the repository construction and the prevailing README, then proposes an inventory of enhancements for customers to approve or reject. An experimental multi-agent conversational mode can also be being developed, permitting customers to specify desired enhancements in free-form pure language by way of the CLI. OSA interprets this request and applies the corresponding adjustments. This mode is at present beneath lively improvement.
One other key energy of OSA is its flexibility with language fashions. It really works with fashionable suppliers like OpenRouter and OpenAI, in addition to native fashions equivalent to Ollama and self-hosted LLMs operating by way of FastAPI.
OSA additionally helps a number of repository platforms, together with GitHub and GitLab (each GitLab.com and self-hosted cases). It could actually alter CI/CD configuration information, arrange documentation deployment workflows, and accurately configure paths for group documentation.
an experimental multi-agent system (MAS), at present beneath lively improvement, that serves as the idea for its computerized and conversational modes. The system decomposes repository enchancment right into a sequence of reasoning and execution levels, every dealt with by a specialised agent. Brokers talk by way of a shared state and are coordinated by way of a directed state graph, enabling conditional transitions and iterative workflows.
README technology
OSA features a README technology software that mechanically creates clear and helpful README information in two codecs: a regular README and an article-style README. The software decides which format to make use of by itself, for instance, if the person gives a path or URL to a scientific paper by way of the CLI, OSA switches to the article format. To begin, it scans the repository to seek out a very powerful information, specializing in core logic and venture descriptions, and takes under consideration the folder construction and any current README.
For the usual README, OSA analyzes the important thing venture information, repository construction, metadata, and the principle sections of an current README if one is current. It then generates a “Core Options” part that serves as the inspiration for the remainder of the doc. Utilizing this data, OSA writes a transparent venture overview and provides a “Getting Began” part when instance scripts or demo information can be found, serving to customers shortly perceive easy methods to use the venture.
In article mode, the software creates a abstract of the related scientific paper and extracts related data from the principle code information. These items are mixed into an Overview that explains the venture objectives, a Content material part that describes the principle parts and the way they work collectively, and an Algorithms part that explains how the carried out strategies match into the analysis. This strategy retains the documentation scientifically correct whereas making it simpler to learn and perceive.
Documentation technology
The documentation technology software produces concise, context-aware documentation for features, strategies, lessons, and code modules. The documentation technology course of is as follows:
(1) Reference parsing: Initially, a TreeSitter-driven parser fetches imported modules and resolves paths to them for every explicit supply code file, forming an import map that can additional be used to find out methodology and performance requires the international modules utility. By implementing such an strategy, it’s comparatively straightforward to rectify interconnections between completely different elements of the processed venture and to tell apart between inside aliases. Together with the import maps, the parser additionally preserves basic data such because the processing file, an inventory of occurring lessons, and standalone features. Every class comprises its title, attributes checklist, decorators, docstring, checklist of its strategies, and every methodology has its particular particulars that are of the identical construction as standalone features, that’s: methodology title, docstring, return sort, supply code and alias resolved international methodology calls with a reputation of the imported module, class, methodology, and path to it.
(2) Preliminary docstrings technology for features, strategies, and lessons: With a parser having a construction fashioned, an preliminary docstrings technology stage is ongoing. Solely docstrings that lack lessons, strategies, and features are processed at this stage. Here’s a basic description of what the ‘what’ methodology does. The context is usually the tactic’s supply code, since at this level, forming a basic description of the performance is essential. The onward immediate contains details about the tactic’s arguments and interior designers, and it trails with the supply code of the referred to as international strategies to offer extra context for processing methodology utility. A neat second right here is that class docstrings are generated solely in spite of everything their docstring-lacking strategies are generated; then class attributes, their strategies’ names, and docstrings are offered to the mannequin.
(3) Technology of “the principle thought” of the venture utilizing descriptions of parts derived from the earlier stage.
(4) Docstrings replace utilizing generated “most important thought”: Therefore, all docstrings for the venture are presumably current, technology of the principle thought of the venture might be carried out. Basically, the immediate for the thought consists of docstrings for all lessons and features, together with their significance rating based mostly on the speed of prevalence of every part within the import maps talked about earlier than, and their place within the venture hierarchy decided by supply path. The mannequin response is returned in markdown format, summarizing the venture’s parts. As soon as the principle thought is acquired, the second stage of docstring technology begins, throughout which the entire venture’s supply code parts are processed. At this second, the important thing focus is on offering the mannequin with an authentic or generated docstring on the preliminary stage docstring with the principle thought to elaborate on ‘why’ this part is required for the venture. The supply code for the strategies can also be being offered, since an expanded venture narrative might immediate the mannequin to appropriate some factors within the authentic docstring.
(5) Hierarchical modules description technology ranging from the underside to the highest.
(6) Utilizing Mkdocs and GitHub pages for automated documentation pushing and streaming: Last stage of the docstring pipeline, contemplating a recursive traversal throughout the venture’s modules and submodules. Hierarchy is predicated on the supply path; at every leaf-processing stage, a beforehand parsed construction is used to create an outline of which submodule is used, in accordance with the principle thought. As processing strikes to increased ranges of the hierarchy, generated submodules’ summaries are additionally used to offer extra context. The mannequin returns summaries in Markdown to make sure seamless integration with the mkdocs documentation technology pipeline. The entire schema of the strategy is described within the picture beneath.
CI/CD and construction group
OSA presents an automatic CI/CD setup that works throughout completely different repository internet hosting platforms. It generates configurable workflows that make it simpler to run exams, test code high quality, and deploy tasks. The software helps frequent utilities equivalent to Black for code formatting, unit_test for operating exams, PEP8 and autopep8 for type checks, fix_pep8 for computerized type fixes, pypi_publish for publishing packages, and slash_command_dispatch for dealing with instructions. Relying on the platform, these workflows are positioned within the acceptable places, for instance, .github/workflows/ for GitHub or a .gitlab-ci.yml file within the repository root for GitLab.
Customers can customise the generated workflows utilizing choices like –use-poetry to allow Poetry for dependency administration, –branches to outline which branches set off the workflows (by default, most important and grasp), and code protection settings by way of --codecov-token and --include-codecov.
To make sure dependable testing, OSA additionally reorganizes the repository construction. It identifies take a look at and instance information and strikes them into standardized exams and examples directories, permitting CI workflows to run exams persistently with out extra configuration.
Workflow information are created from templates that mix project-specific data with user-defined settings. This strategy retains workflows constant throughout tasks whereas nonetheless permitting flexibility when wanted.
OSA additionally automates documentation deployment utilizing MkDocs. For GitHub repositories, it generates a YAML workflow within the .github/workflows listing and requires enabling learn/write permissions and deciding on the gh-pages department for deployment within the repository settings. For GitLab, OSA creates or updates the .gitlab-ci.yml file to incorporate construct and deployment jobs utilizing Docker photographs, scripts, and artifact retention guidelines. Documentation is then mechanically revealed when adjustments are merged into the principle department.
How one can use OSA
To start utilizing OSA, select your repository with draft code that’s incomplete or underdocumented. Optionally, embody a associated scientific paper or one other doc describing the library or algorithm carried out within the chosen repo. The paper is uploaded as a separate file and used to generate the README. It’s also possible to specify the LLM supplier (e.g., OpenAI) and the mannequin title (equivalent to GPT-4o).
OSA generates suggestions for bettering the repository, together with:
- A README file generated from code evaluation, utilizing normal templates and examples
- Docstrings for lessons and strategies which are at present lacking, to allow computerized documentation technology with MkDocs
- Fundamental CI/CD scripts, together with linters and automatic exams
- A report with actionable suggestions for bettering the repository
- Contribution pointers and information (Code of Conduct, pull request and concern templates, and many others.)
You possibly can simply set up OSA by operating:
pip set up osa_toolAfter organising the surroundings, you need to select an LLM supplier (equivalent to OpenAI or an area mannequin). Subsequent, you need to add GIT_TOKEN (GitHub token with normal repo permissions) and OPENAI_API_KEY (for those who use OpenAI-compatible API) as surroundings variables, or you possibly can retailer them within the .env file as properly. Lastly, you possibly can launch OSA immediately from the command line. OSA is designed to work with an current open-source repository by offering its URL. The essential launch command contains the repository handle and non-compulsory parameters such because the operation mode, API endpoint, and mannequin title:
osa_tool -r {repository} [--mode {mode}] [--api {api}] [--base-url {base_url}] [--model {model_name}]OSA helps three working modes:
- auto (default) – analyzes the repository and creates a personalized enchancment plan utilizing the specialised LLM agent.
- primary – applies a predefined set of enhancements: generates a venture report, README, group pointers, an “About” part, and creates normal directories for exams and examples (if they’re lacking).
- superior – permits handbook choice and configuration of actions earlier than execution.
Further CLI choices can be found right here. You possibly can customise OSA by passing these choices as arguments to the CLI, or by deciding on desired options within the interactive command-line mode.
As soon as launched, OSA performs an preliminary evaluation of the repository and shows key data: basic venture particulars, the present surroundings configuration, and tables with deliberate and inactive actions. The person is then prompted to both settle for the recommended plan, cancel the operation, or enter an interactive enhancing mode.
In interactive mode, the plan might be modified: actions toggled on or off, parameters (strings and lists) adjusted, and extra choices configured. The system guides the person by way of every motion’s description, attainable values, and present settings. This course of continues till the person confirms the ultimate plan.
This CLI-based workflow ensures flexibility, from totally automated processing to specific handbook management, making it appropriate for each fast preliminary assessments and detailed venture refinements.
OSA additionally contains an experimental conversational interplay mode that permits customers to specify desired repository enhancements utilizing free-form pure language by way of the CLI. If the request is ambiguous or insufficiently associated to repository processing, the system iteratively requests clarifications and permits the hooked up supplementary file to be up to date. As soon as a legitimate instruction is obtained, OSA analyzes the repository, selects the suitable inside modules, and executes the corresponding actions. This mode is at present beneath lively improvement.
When OSA finishes, it creates a pull request (PR) within the repository. The PR contains all proposed adjustments, such because the README, docstrings, documentation web page, CI/CD scripts, сontribution pointers, report, and extra. The person can simply overview the PR, make adjustments if wanted, and merge it into the venture’s most important department.
Let’s have a look at an instance. GAN-MFS is a repository that gives a PyTorch implementation of Wasserstein GAN with Gradient Penalty (WGAN-GP). Right here is an instance of a command to launch OSA on this repo:
osa_tool -r github.com/Roman223/GAN_MFS --mode auto --api openai --base-url https://api.openai.com/v1 --model gpt-4.1-miniOSA made a number of contributions to the repository, together with a README file generated from the paper’s content material.
OSA additionally added a License file to the pull request, in addition to some primary CI/CD scripts.
OSA added docstrings to all lessons and strategies the place documentation was lacking. It additionally generated a structured, web-based documentation website utilizing these docstrings.
The generated report contains an audit of the repository’s key parts: README, license, documentation, utilization examples, exams, and a venture abstract. It additionally analyzes key sections of the repository, equivalent to construction, README, and documentation. Primarily based on this evaluation, the system identifies key areas for enchancment and gives focused advice.
Lastly, OSA interacts with the goal repository by way of GitHub. The OSA bot creates a fork of the repository and opens a pull request that features all proposed adjustments. The developer solely must overview the ideas and alter something that appears incorrect. In my view, that is a lot simpler than writing the identical README from scratch. After overview, the repository maintainer efficiently merged the pull request. All adjustments proposed by OSA can be found right here.
Though the variety of adjustments launched by the OSA is important, it’s troublesome to evaluate the general enchancment in repository high quality. To do that, we determined to look at the repository from a safety perspective. The scorecard software permits us to judge the repository utilizing the aggregated metric. Scorecard was created to assist open supply maintainers enhance their safety finest practices and to assist open supply customers decide whether or not their dependencies are secure. The combination rating takes under consideration many repository parameters, together with the presence of binary artifacts, CI/CD exams, the variety of contributors, and a license. The aggregated rating of the unique repository was 2.2/10. After the processing by OSA, it rose to three.7/10. This occurred because of the addition of a license and CI/CD scripts. This rating should appear too low, however the repository being processed isn’t meant for integration into giant tasks. It’s a small software for producing artificial information based mostly on a scientific article, so its safety necessities are decrease.
What’s Subsequent for OSA?
We plan to combine a RAG system into OSA, based mostly on finest practices in open-source improvement. OSA will evaluate the goal repository with reference examples to determine lacking parts. For instance, if the repository already has a high-quality README, it received’t be regenerated. Initially, we used OSA for Python repositories, however we plan to help extra programming languages sooner or later.
If in case you have an open repository that requires enchancment, give OSA a attempt! We’d additionally respect concepts for brand spanking new options you could go away as points and PRs.
For those who want to use OSA in your works, it may be cited as:
Nikitin N. et al. An LLM-Powered Software for Enhancing Scientific Open-Supply Repositories // Championing Open-source DEvelopment in ML Workshop@ ICML25.
