E3.2: Bringing Agentic to Life

Deploy an AI Workflow to Scrape Reddit, Distill valuable Information, and Autonomously Post Findings on Twitter.

Agentic operating fully autonomously on X

Introduction

In our previous blog, we explored the differences between production and non-production environments. Now, it's time to take that knowledge a step further with a practical implementation. In this post, we'll deploy a Reddit scraper that leverages Large Language Models (LLMs) and eventually posts its findings on Twitter in a fully automatic way, running once daily. Due to past experience, we'll use Microsoft Azure and VS Code for this deployment. Yet you can use a Cloud provider of choice as such deployments will be more or less similar among Clouds.

Why Learning Cloud Deployment is Crucial

Understanding cloud deployment is essential for modern software development. Moving your projects to a cloud environment brings numerous benefits, including scalability, reliability, and accessibility. It also equips you with skills that are highly valued in the industry. In this blog, we will take the Twitter scraper from blog E2.2 and deploy it, demonstrating the process step-by-step.

Recap of the System

Agentic consists of two agents: a Reddit researcher and an editor-in-chief. We scrape information of posts and comments on the subreddits specified in the variables.py file:

The researcher gathers information from Reddit, and the editor checks the work of the researcher. Once done, findings are posted to Twitter. We use open-source Agentic Workflow framework Crew AI for this. For more information, you can check the GitHub repository and the previous blog post.

Functional Requirements for Deployment

Let’s use the framework as outlined in the previous blog to better understand how to approach the deployment.

Understanding the Production Environment

• Scalability Requirements:
  The system is designed to run locally with limited RAM and needs to operate only once per day, making extensive scalability unnecessary. A cron job will be used to schedule this daily task. A cron job is a way to set up automatic tasks on a computer to run at specific times, like setting an alarm clock for your software.

• Integration Points:
  As a new deployment, there are no existing systems to integrate with.

• Performance Constraints:
  Given the simplicity of this deployment, performance constraints are minimal.

Code, Versioning, and CI/CD Pipelines

• Version Control Systems (VCS):
  We'll use GitHub for version control, as it provides a free and robust platform for managing code.

• Model Versioning:
  Ensure the deployed version matches the most recent version on GitHub. The only exception is the pysqlite3-binary requirement and corresponding code in function_app.py, necessary for Linux deployment. Make sure to comment it out when running locally on Windows.

• CI/CD Tools:
  While CI/CD tools can be beneficial, they are overkill for this simple deployment.

• Pipeline Stages:
  For this project, automated pipelines are not required.

Containerisation and Orchestration

Containerisation and orchestration are not necessary for this small-scale deployment.

Monitoring and Logging

• Logging Practices:
  Implementing different logging levels is crucial for monitoring the bot's performance and debugging any issues. Azure's portal provides comprehensive tools for capturing and analyzing logs. Using the logging capabilities within the Azure function app suffices here.

Data and Model Governance

Not applicable for this hobby project.

Security Considerations

• API Keys:
  Store API keys in a .env file to follow best practices for security and configuration management.

Performance Optimisation

Performance optimisation is not a priority due to the deployment's simplicity.

Documentation and Knowledge Sharing

Sharing your deployment experience through a blog post is an excellent way to document the process and help others. :)

The Deployment Process

1. Download and install the prerequisites

   1. Check out the Microsoft website for deployment of functions using VS code.

      1. Specifically, download the Core Tools and the Azure Functions extension.

2. Clone the repository from GitHub and set up your local environment.

3. Set up your .env file with your API keys.

4. If you don’t want to setup the Azure functions yourself and leverage my files, go to step 8. If you want to do it yourself, copy research_tweeting and only keep main_content.py, custom_functions.py, requirements.txt and variables.py and perform the steps below.

5. Create a Local Function Project:
   Use Azure Functions to create a Python-based function project with a time trigger.

6. Go through the steps and finally set the Cron Job:
   Configure the cron job to run the function daily at 1 PM:

The default setting is that the system also runs at startup. If not desired, change it in function_app.py.

7. Directory Structure:
   A lot of files are automatically being created now. Ensure your project directory is correctly structured. At this point, you Azure function files are at the same level as the research_tweeting folder. Unfortunately, your gitignore and requirements.txt function may be overwritten by Azure. Check this. Make sure to add the right lines back.

8. Local Function Testing:

   1. Use Azurite for local testing. Hit crl + Shift + P and Start Azurite.

   

   1. UTF Encoding:
      When you’re on a Windows, you will get an UTF encoding error. You can fix this by running Set $env:PYTHONIOENCODING = "utf-8" for proper encoding.

   2. Run the Function Locally:
      Execute func start --verbose to test your function locally (the verbose flag is for verbose logging).

9. Deploy to Azure:

   1. Create a resource group and storage account on Azure.

   2. Run your version of the following command with your resource group and storage account you just made:

   az functionapp create --resource-group agentic --consumption-plan-location westeurope --runtime python --os-type Linux --functions-version 4 --name agentic-twitter-app --storage-account agenticstorage

10. Publish to Azure

    1. Run the following command to publish your app to the app you just created. For any code changes, you can run this command as well.

func azure functionapp publish agentic-twitter-app

5. Check Logs:
   Monitor the logs in the agentic-twitter-app function app via the Azure portal which you can find under the section Monitoring within your function app. If you see you’re Crew AI logs, you’re up and running and your crew will start posting on Twitter soon.

Agentic on X

Conclusion

Deploying a bot like Agentic to a cloud environment like Azure demonstrates the practical application of cloud deployment skills. This process not only enhances the functionality and reliability of your projects but also equips you with essential knowledge for real-world applications. By following this guide, you can confidently take your projects from local development to production, ensuring they deliver value effectively and efficiently.

Call to Action

Ready to deploy your own projects? Leverage the approach in this blog to set up your own scraper that posts to Twitter. Embrace the power of cloud deployment and see how it transforms your workflow. The full code can be found in the GitHub repository.

Thanks and enjoy coding!

Michiel

Please reach out to chat about potential applications of the system.