story-research-zapwall/fixKnowledge/ssh-connection-errors-deployment.md

SSH Connection Errors During Deployment

Date: 2024-12-19 Auteur: Équipe 4NK

Problem Description

During deployment, SSH connection errors occurred when verifying the Git repository on the server. The errors were:

mux_client_request_session: read from master failed: Connection reset by peer
Failed to connect to new control master
mm_send_fd: sendmsg(2): Broken pipe
mux_client_request_session: send fds failed

These errors appeared at step 5 of the deployment script when checking if Git is initialized on the server.

Additionally, the script could hang indefinitely at step 5 due to command substitution blocking when SSH connections failed or timed out.

Impact

• Severity: Medium
• Scope: Deployment script reliability
• User Impact: Deployment could fail or continue with errors, potentially leaving the server in an inconsistent state
• Frequency: Intermittent, occurring when SSH ControlMaster connection is interrupted

Root Cause

The SSH ControlMaster multiplexing connection was being closed prematurely or becoming stale, causing subsequent SSH commands to fail. The original ssh_exec function did not handle connection failures robustly:

1. No connection validation: The function did not check if the ControlMaster socket was still valid before use
2. No retry mechanism: Failed connections were not retried after cleanup
3. No dead connection cleanup: Stale connections were not detected and cleaned up before reuse
4. Silent failures: Connection errors in conditional checks could be misinterpreted as command failures

Root Cause Analysis

The SSH ControlMaster feature creates a persistent connection to avoid multiple SSH handshakes. However:

• Network interruptions can close the master connection
• The ControlMaster socket file can become stale if the connection dies
• The script did not detect or handle these cases, leading to cascading failures
• Command substitution $() with SSH commands can block indefinitely if the connection hangs
• The original logic used grep on command output, which required capturing all output and could hang

Deep Root Cause: ControlSocket Already Exists

The fundamental issue is that:

1. Socket can become invalid between check and execution: The socket file can exist on the filesystem, but the underlying connection can die between the validation check and the actual command execution.

2. ControlMaster=auto behavior: When SSH detects that a ControlMaster socket exists but is invalid, it does NOT remove the socket. Instead, it:

   • Disables multiplexing for that command
   • Creates a new connection without multiplexing
   • Leaves the dead socket file in place

3. Dead socket accumulation: The dead socket remains and causes subsequent commands to fail with "ControlSocket already exists, disabling multiplexing" errors.

4. Insufficient cleanup: Previous cleanup attempts only removed the socket file, but if a process was still holding it, the socket could remain. Removing the entire directory ensures complete cleanup.

The solution is to:

• Test actual connection usability (not just socket existence)
• Aggressively remove the entire socket directory when invalid
• Recreate the directory for fresh connections

Corrections Applied

1. Enhanced SSH Connection Management

File: deploy.sh

Changes:

• Added cleanup_dead_ssh() function to properly clean up dead SSH connections
• Added check_ssh_connection() function to validate ControlMaster connection before use
• Enhanced ssh_exec() function with:
  • Connection validation before each command
  • Automatic cleanup of dead connections
  • Additional SSH options for better connection stability:
    • ConnectTimeout=10: Fail fast if connection cannot be established
    • ServerAliveInterval=60: Keep connection alive
    • ServerAliveCountMax=3: Detect dead connections quickly

Latest improvements (root cause fix):

• check_ssh_connection() now tests actual connection usability with ssh ... true instead of just ssh -O check
• cleanup_dead_ssh() now removes entire socket directory instead of just socket file
• This ensures socket is truly removed even if process is still holding it
• Directory is recreated after cleanup for fresh connections

2. Improved Error Handling at Step 5

File: deploy.sh

Changes:

• Enhanced Git repository verification to properly handle SSH connection errors
• Added explicit error detection and recovery mechanism
• Added automatic retry after connection cleanup
• Better error messages to distinguish between connection errors and Git initialization needs
• Fixed blocking issue: Removed command substitution $() that could hang indefinitely
• Simplified logic to use direct exit code checking instead of parsing output
• Improved timeout handling to prevent script from hanging

Modifications

Files Modified

• deploy.sh:
  • Enhanced ssh_exec() function with retry logic and connection validation
  • Added cleanup_dead_ssh() and check_ssh_connection() helper functions
  • Improved error handling in Git repository verification step

Code Changes

Before:

ssh_exec() {
    ssh -o ControlMaster=auto -o ControlPath="${SSH_CONTROL_PATH}" -o ControlPersist=300 ${SERVER} "$@"
}

After:

ssh_exec() {
    # Validates connection, cleans up dead connections, retries on failure
    # Includes connection stability options
}

Deployment Procedures

Automatic Deployment

The fix is automatically applied when using the deployment script:

./deploy.sh "commit message"

No manual intervention required. The script now handles SSH connection errors automatically.

Verification

After deployment, verify that SSH connections are stable:

1. Check that deployment completes without SSH errors
2. Monitor for connection errors in subsequent deployments
3. Verify that retry mechanism works correctly

Analysis Procedures

Monitoring SSH Connection Issues

1. Check deployment logs for SSH connection errors:

   # Review recent deployment output
   

2. Verify SSH ControlMaster socket:

   # On the deployment machine
   ls -la /tmp/ssh_control_*/
   

3. Test SSH connection manually:

   ssh -O check -o ControlPath="/tmp/ssh_control_*/debian_92.243.27.35_22" debian@92.243.27.35
   

Debugging Steps

If SSH connection errors persist:

1. Check network connectivity to the server
2. Verify SSH server configuration allows ControlMaster
3. Check for firewall or network issues
4. Review SSH server logs on the remote machine
5. Verify SSH key authentication is working

Logs to Review

• Deployment script output (stdout/stderr)
• SSH client logs (if verbose mode enabled)
• Remote SSH server logs: /var/log/auth.log or similar

Prevention

Best Practices

1. Connection validation: Always validate SSH connections before use
2. Retry logic: Implement retry mechanisms for network operations
3. Cleanup: Properly clean up stale connections
4. Error handling: Distinguish between different types of failures
5. Monitoring: Monitor connection stability over time

Future Improvements

• Add connection health metrics
• Implement exponential backoff for retries
• Add connection pooling if needed
• Consider alternative connection methods if ControlMaster proves unreliable

Related Issues

None identified at this time.

References

• SSH ControlMaster documentation
• Deployment script: deploy.sh
• Related documentation: docs/deployment.md