Testing Login, Roles and Data Access in an AI-Built App

To test login and roles in an AI-built app, write a matrix of what each role may see and change, then test every cell, including the "must refuse" ones, with two accounts per role. Add five checks AI tools often miss: users cannot change their role, sign-up is as closed as intended, admin checks run on the server, data rules hold without a session, and old sessions really end.

If you would rather have your app's access control tested for you, see how RAITHub would test this below.

Broken access control is A01, the top risk, in the OWASP Top 10:2025. AI tools make it easy to ship: they build the screens each role should see, and the screens look right. What they do not do is log in as the wrong person and try the door. This post is a defensive plan for testing an app you own or have written permission to test. If you already suspect a leak, start with users can see other tenants' data, which covers the first hour and the fix.

What goes wrong with login and roles in AI-built apps?

Public research on vibe-coded apps keeps finding the same five patterns:

Pattern	Evidence	The test
Login logic in the browser	Wiz found passwords hard-coded in JavaScript and login flags kept in localStorage (Wiz Research, 2025)	Call protected APIs directly, with no browser involved
Missing or loose database rules	Misconfigured row-level security headed Escape's findings across 5,600 apps (Escape, 2025)	Two-account and no-session reads on every table
Roles stored where users can edit them	Supabase warns that user metadata can be updated by the signed-in user and is not for authorisation data (Supabase RLS guide)	Try to change your own role as a normal user
Sign-up open when it should be closed	Wiz found a platform flaw that let anyone register on private Base44 apps using only a public app ID (Wiz, July 2025); it was fixed within 24 hours	Register through the API, not only the form
Access checked only in middleware or the UI	CVE-2025-29927 let requests skip Next.js middleware on unpatched versions (Next.js advisory)	Check permissions where data is read, and test the API routes directly

How do you build a role matrix?

One row per action, one column per role, and an expected result in every cell. Include anonymous visitors as a role. A small SaaS might look like this:

Action	Anonymous	Member	Team admin	Platform admin
Read own team's projects	Refuse	Allow	Allow	Allow
Read another team's projects	Refuse	Refuse	Refuse	Allow, logged
Invite a teammate	Refuse	Refuse	Allow	Allow
Change own role or plan	Refuse	Refuse	Refuse	Allow
View billing and invoices	Refuse	Refuse	Allow, own team	Allow
Open /admin and its APIs	Refuse	Refuse	Refuse	Allow

Most of the real bugs live in the "Refuse" cells, which is exactly where nobody clicks during a demo. Write the matrix before testing; without it you can only check that the app does what the code does. Ask your AI tool to draft it from the code if you like, then correct it by hand: the point is what you intended, not what was built.

How do you test every cell?

Create two accounts for each role, in two different teams. For each row, make the request as each role, through the API rather than the screens, because the screens hide what the API will still allow. OWASP calls the cross-account case broken object level authorization (OWASP API1:2023). A Playwright test that loops over the matrix keeps it running after every prompt:

import { test, expect } from '@playwright/test'

const tokens = {
  anonymous: '',
  member: process.env.MEMBER_TOKEN,
  teamAdmin: process.env.TEAM_ADMIN_TOKEN,
}

// A project that belongs to a different team from both test users
const otherTeamProject = '/api/projects/' + process.env.OTHER_TEAM_PROJECT_ID

for (const [role, token] of Object.entries(tokens)) {
  test(role + ' cannot read another team project', async ({ request }) => {
    const res = await request.get(otherTeamProject, {
      headers: token ? { Authorization: 'Bearer ' + token } : {},
    })
    expect([401, 403, 404]).toContain(res.status())
  })
}

Add one loop per "Refuse" row. For writes, also check the data afterwards: some APIs return 403 but still change the record.

Can a user make themselves an admin?

This failure is easy to miss, because nothing on screen looks wrong. Two versions to test for:

• Role in user metadata. The app reads the role from the user's own metadata, which the user can update from the browser. Supabase's guide is explicit: raw_user_meta_data "can be updated by the authenticated user" and "is not a good place to store authorization data"; raw_app_meta_data cannot be updated by the user (Supabase RLS guide).
• Role column on an editable profile. A policy such as "users can update their own profile" covers the whole row, including a role or plan column.

On a Supabase or Postgres project you can test the database rules directly, as a specific user, inside a transaction that is rolled back. Run it on a staging copy, not production:

begin;
-- Act as user B, a normal member
set local role authenticated;
select set_config('request.jwt.claims',
  '{"sub":"USER_B_UUID","role":"authenticated"}', true);

-- 1. Can B read A's invoices? Expect 0.
select count(*) from invoices where user_id = 'USER_A_UUID';

-- 2. Can B promote themselves? Expect an error or "UPDATE 0".
update profiles set role = 'admin' where id = 'USER_B_UUID';

-- 3. Can B move a project into another team? Expect an error or "UPDATE 0".
update projects set team_id = 'OTHER_TEAM_UUID' where owner_id = 'USER_B_UUID';
rollback;

Replace the placeholders with real test-account IDs. If any line succeeds, fix the policy, for example with a separate roles table that only the server writes, or a column-level grant. The Postgres row-level security guide covers policy patterns, and RLS disabled in public covers tables with no rules at all.

Is sign-up as closed as you think?

• If the app is invite-only or internal, call the sign-up endpoint directly with a new email. Disabling the form is not the same as disabling sign-up in the auth provider.
• If sign-up should be limited to your company's domain, try an outside address, and an address that only looks similar.
• Check that an unverified account cannot reach anything a verified one can.
• Check that an invite link works once, for the invited email only, and expires.

Do sessions and password resets end when they should?

OWASP's Forgot Password Cheat Sheet sets the bar: reset links are single use and expire, the reset page gives the same message whether or not the account exists, and a successful reset invalidates existing sessions. Test each:

• Use a reset link twice. The second use must fail.
• Request a reset for an email with no account. The message must match the one for a real account.
• Log in on two browsers, reset the password in one, and refresh the other. It should be logged out.
• Log out, then replay a request with the old token from the developer tools. It should be refused once the token expires, and your token lifetime should be short enough that this matters.
• Remove a teammate, then check they lose access on their next request, not at their next login.
• Try 20 wrong passwords in a minute. Something should slow you down without locking the real owner out for good.

Where should the admin check live?

On the server, next to the data, for every request. Middleware, layout guards and hidden menus are useful for the user experience, but each can be skipped. CVE-2025-29927 is the reminder: the Next.js advisory says it was possible to bypass authorisation "if the authorization check occurs in middleware" (Next.js advisory). Update Next.js, and also confirm every admin API route checks the role itself. When Next.js middleware is not running covers a related failure.

How long does this take to test yourself?

For an app with three roles and a dozen data endpoints, plan 1–2 days for a developer who knows the stack: a couple of hours for the matrix and test accounts, a day for the API and database tests, and a few hours for sign-up and sessions. The main risk of doing it yourself is testing only the cells you expect to fail. Test every "Refuse" cell, even the ones that seem obvious.

What should you test first on a small budget?

The cross-account read on your most sensitive table, the self-promotion test, and every admin API without a session. Those three take an afternoon and cover the failures that expose other people's data or hand over the whole app. The broader security list is in the vibe-coded app security checklist, and the full pre-launch list in the AI-built SaaS launch checklist.

Buy, build or hire?

Option	Choose this when	Trade-off
Managed auth with built-in roles (an auth provider's organisations and roles features)	Your roles are simple and you have not built custom ones yet	Less code to get wrong; your data rules still need testing
A tool or SaaS testing platform (scanners plus Playwright)	A developer can own the matrix and keep the tests current	Scanners find open endpoints; they cannot know your role rules
Freelancers or crowdtesting	You want a quick outside look before launch	Variable depth on authorisation; check the tester's security experience
An in-house QA or security hire	Roles and permissions change often in a growing product	Strong ownership; slow and costly for an early app
A managed QAaaS team or a one-off security and launch audit	You want the matrix tested end to end before real users	An outside dependency; make sure the tests stay in your repo

For the difference between this kind of testing and a formal pentest, see pentest vs vulnerability scanning.

Why RAITHub for this

• Access control is everyday work. Sundor Skin, a B2B wholesale platform RAITHub built, has 146 PostgreSQL tables under row-level security, 88 permission codes and 12 staff roles, with 530+ tests. PropDesk has 4 roles and 1,024 tests.
• Tests at the database, the API and the screen. The matrix is tested where the data is enforced, not just where it is displayed.
• Honest limits. RAITHub's security testing is application-level testing against OWASP guidance. It is not a CREST- or PCI-certified penetration test and produces no compliance attestation. There is no AI-built-app case study yet.

When you don't need us

• Your app has one role and no shared data, for example a single-user tool. Test the no-session read and the reset flow yourself.
• A developer who did not build the app can give it two days with this plan.
• A customer or regulator requires a certified penetration test report. Use a certified testing firm.

How RAITHub would test this

• Scope: agree the roles, the sensitive data and the auth provider on a free 15-minute call; you confirm in writing that you own the app or are authorised to have it tested.
• Matrix: draft the role matrix from your intent and the code, and agree it with you.
• Test: every cell through the API with two accounts per role, database policy tests on a staging copy, sign-up, invites, resets and sessions.
• Report: each finding ranked by data at risk, with reproduction steps and a suggested fix, plus the matrix as automated tests in your repository.
• What you receive: the report, the tests and a handover note. IP is yours; an NDA is standard.

The audit is fixed-price, set in a written quote after the call. See AI-built app testing, security testing and the QA as a service overview, or ask for an access-control and launch audit.

Frequently asked questions

How do I test user roles in an app built with AI?

Write a matrix of every action against every role, including anonymous visitors, then test each cell through the API with two accounts per role. Focus on the cells that should be refused, and automate them so the next prompt cannot quietly reopen them.

Where should I store user roles in Supabase?

Somewhere users cannot write: app metadata, or a roles table that only your server or an admin can change. Supabase's guide warns that user metadata can be updated by the signed-in user, so it should not drive authorisation.

Is hiding admin buttons enough to protect admin features?

No. Anyone can call your API directly. Every admin route and API must check the user's role on the server, at the point where data is read or changed.

Should a refused request return 403 or 404?

Either is acceptable for another user's record. Many teams return 404 so the response does not confirm the record exists. What matters is that no data is returned and nothing changes.

Is this the same as a penetration test?

No. This is functional and application-level security testing of your own access rules. A penetration test is a broader, often certified engagement. If a customer or auditor needs a pentest report, you need a certified provider.

Can I test my app's login security myself?

Yes, on an app you own or are authorised to test, using test accounts and a staging copy of the data. Follow the matrix, the self-promotion test, the sign-up checks and the OWASP reset rules in this post.