Updated the UI and the backend to send to the end user the scaler config information, this way the user can predict the assumed behavior

per worker rather then scale if pending_jobs_count > 5)

reviewed the frontend code, added some comments and notes, and a bit of logic change between the frontend and the backend, now we do not have the noisy 404 error in the UI console when the requested job is still in queue, we instead return 200 with a payload containing status=queued

review the worker's script, refactor, added some comments and notes, retested the whole system, everything works just fine !

Added some notes/todos for later on.
Small refactoring and retested the whole system. Everything is still working just fine!

Updated the frontend to properly handle the new jobs and stopped using useRef since it created a decoupled state from the react default
component state.
Updated the backend to add the scaler and enhanced the logic of scaling down the workers to check if the cluster is idle (no tasks in progress)
Updated the worker to log to a new channel of the redis logs informing the scaler that he is currently busy, thus the scaler will not kill him (scale down)

About to create a todo.txt file that will layout the upcoming changes

added an automatic scaler for our workers, it is basically a polling loop that will check every now and then how many jobs are there in the redis queue, if #>maxthreshold then scale up if #<minthreshold then scale down. min and maxthreshold are diffrent values to not be stuck in a pivot value leading to instant and reoccurent scaling up and down ops

added .env to frontend that saves the API_URL, removed the vite proxy since it was not working proprely once compiled and server via ngnix

first draft of our docker-compose file (stack.yml), this is what dictactes how the whole project will run as one peice