(Environment) (AWS, Mac) Project Execution and Modeling in an Enterprise Setting

I have resumed engineering and modeling tasks, which I had temporarily set aside to focus on project management and new business planning.

For a project that involves optimizing (tuning) individual models for image and text processing, as well as integrating multiple models—a skill I honed during my Ph.D. program—I now require a complex and well-structured work environment.

To ensure optimal efficiency under the given constraints, I plan to set up, refine, and document the work environment for future reference.

"If I had eight hours to chop down a tree, I would spend six hours sharpening the axe."

– Abraham Lincoln (1861–1865), the 16th President of the United States

• Review of Work Environment

  1. Managing Work on an Unstable Server & Data Backup

  ✅ Backup & Record Management: Using SFTP to store and synchronize data in a structured manner.
  ✅ GitHub Usage: Limited to baseline model development and version control for key components.
  ✅ Code & Data Management:

  • Initially, using SFTP Sync to manage code and data across the team until the environment stabilizes.
  • Ensuring team members’ PCs are synced for redundancy and preventing data loss.

  ---

  2. Remote Access Setup

  ✅ Primary: ssh (Company Server, AWS)
  ✅ Secondary: ngrok (For remote access to Google Colab)

  🚧 Corporate Security Policy Issues:

  • Internal servers can connect to other internal servers, but external servers cannot access internal servers via SSH.
  • This likely stems from security policies restricting inbound SSH connections.
  • AWS is technically external, so SSH should work unless explicitly blocked for security, cost, or management reasons.

  ---

  3. Personal Work Environment

  ✅ Hardware: Apple Mac M1 Pro (2020)
  ✅ Development Tools:

  • VS Code Extensions: SFTP, SSH, Python, Jupyter Notebook, Beautify, JSON
  • Jupyter Notebook: Main local development environment
  • Google Colab: For cloud-based experiments

  ---

  4. Storage & Data Accessibility

  ✅ Company Internal Server

  • Access to image and text datasets for analysis
  • GPU-enabled servers available for model training

  ✅ Apple Mac M1 Pro as an SFTP Server

  • Acts as a sync server to store backups and mitigate GPU server instability
  • Each unit member’s PC will archive and synchronize important work

  ✅ Google Drive (Temporary Storage)

  • Large dataset & image storage
  • Collaboration with external teams for data exchange
  • Code storage & sharing, ensuring project continuity

  ---

  5. Version Control & History Management

  ✅ Git History installed on both the server and team members’ PCs to maintain commit logs and traceability.

  ---

  6. Collaboration & Communication

  ✅ Messaging Platform: Slack

  • Used for daily task tracking and logging issues within the unit.
  • Chosen over internal company messengers to facilitate future external collaboration.
  • Slack is integrated with GitHub, making it valuable for monitoring baseline models and code updates.
  • Team members must use company emails for Slack accounts.

  ---

  Summary & Next Steps

  🔹 Set up and validate SSH access to AWS (ensure company policy allows it).
  🔹 Refine SFTP sync strategy to improve data redundancy and workflow efficiency.
  🔹 Test Google Drive integration for seamless external collaboration.
  🔹 Monitor and adjust Slack usage as the team grows and external partners join.

  This setup balances security, flexibility, and efficiency, ensuring smooth project execution in a corporate environment with strict security policies and infrastructure limitations. 🚀