Auditing Deployment Controls, Access and Change Management

Deployment is where models move from development environments to production and start making real decisions about real users. The controls required at this stage are different from development; they focus on who can deploy, who can access the system, and how changes are managed. This lesson covers deployment audits: access controls, versioning, change procedures, and the separation between development and production environments.

Access Controls and Authentication

Start by understanding who can access the production AI system and what they can do. In a well-managed environment, access to production models should be restricted to authorised personnel. Common roles include model owner, data analyst and engineer, though each role should have different permissions. A model owner should be able to monitor performance and see decision logs, but not change model parameters. An engineer should be able to deploy updates, though that privilege should be separate from the privilege to bypass approval gates.

Audit access controls by requesting an access list. Who has administrative access to the production model? Who can modify the model, and what approval is required? What audit trail exists for access? Many organisations have poor access control hygiene: people who left months ago still have access, or access was granted informally for a one-off task and never revoked. Test access controls by spot-checking: pick an employee and verify they actually have only the access they should have. Request change logs and verify that every access grant and revocation is recorded with a reason and an approver.

Environment Separation

Production AI should not run on the same infrastructure, with the same data, or with the same permissions as development. An engineer testing a new model in development should not be able to accidentally push it to production, view production data, or modify production model parameters. Many organisations start with development and production on the same server or in the same cloud environment with weak boundaries.

Audit for environment separation by asking: are development, testing, and production environments separate? Can someone with access to development access production? Are different data sets used? Are different approval processes required for changes? A good control is a policy that says production model changes require written approval before deployment, while development allows changes freely.

Versioning and Rollback Capability

Older versions of the model should be available so that if a new version performs badly, the system can roll back to a previous version. This is an important recovery control. Audit for versioning by asking: can you retrieve any prior version of the model? How many versions are kept? If the organisation only keeps the current version, they have no rollback capability.

Test this practically: request a model from six months ago. Can they produce it? How long does it take? If they cannot, or if it takes a week, then the control is not effective. A working rollback control should let you revert to a prior version in minutes, not days.

Change Procedures

Every change to a production model should follow a documented procedure: request, review, approval, deployment, and monitoring. Audit by sampling changes. Pull change requests for the most recent ten model updates. For each: Did a change request exist before the deployment? Was it reviewed by someone other than the requester? Was it approved by an authorised person? What monitoring happened after deployment to confirm the change worked as intended?

You will often find that changes happened without formal requests, or requests were approved after the change was deployed to cover up a shortcut. If formal procedures exist but are not followed, you need to flag why that happened. Is the procedure too cumbersome? Are approvers not available fast enough? Understanding barriers to compliance helps you recommend controls that are realistic as well as effective.