Loading memory…
Loading memory…
Design and implement new data pipelines tailored to a specific project or new product. Continuously improve data pipelines to make them more generic and modular, reducing the effort required for subsequent integrations. Write, optimise and refactor prompts and prompt-related pipelines. Conduct thorough testing and validation of data pipelines to ensure accuracy and consistency of data. Ensure pipelines are optimized for performance, scalability, and reusability to facilitate future projects. We want to support people’s enrichment within the CatchAll Tool. Do research on what is available on the market that can help us enrich data, Double-check open databases, and verify whether there is already an existing solution available within the NewsCatcher’s code base. Plan and design the pipeline. Build a prototype and demo it to the whole team. Proceed to, test on Dev, write test cases around. Deploy on Prod, get feedback and improve. One of the users complained about a job being stuck. Debug the pipeline, try to understand where the problem comes from: promts, code, database. Find the issue and prepare a fix. Test it within the whole pipeline on Dev. Break some pipeline, fix and test aga