Data Flow Architecture

This page describes how data moves through the iAm system, from user interaction to persistent storage and back.

High-Level Data Flow

Mobile/Web Client
    │
    │  GraphQL Queries/Mutations
    ▼
Apollo Server (Express)
    │
    │  Neo4j GraphQL OGM
    ▼
Neo4j Database

Additional paths:

• Real-time: Client connects via WebSocket for live session subscriptions
• Background jobs: Server dispatches to Redis/BullMQ for async processing
• Local storage: Client maintains a SQLite database for offline-first operation
• AI services: Server calls external AI APIs for analysis features

The Reporting Process

When an observer reports an experience during a session, the following sequence occurs:

1. Observer begins a Session implementing a specific SessionTypeVersion
2. As experiences arise, Experience Instances (Manifestation nodes) are created with precise timestamps:
   • beginInputTimestamp -- when the observer starts responding
   • endInputTimestamp -- when the observer finishes input
   • submitInputTimestamp -- when the experience instance is submitted
3. Each Experience Instance links to a Unique Value (creating or finding existing)
4. Each Unique Value links to the relevant ExperienceTypeVersion
5. Aggregates may group simultaneous Experience Instances (Manifestation nodes) that co-occur
6. Session ends with post-notes and metadata

GraphQL Flow Example

Client sends mutation:

mutation CreateManifestation($input: CreateManifestationInput!) {
  createManifestation(input: $input) {
    id
    value
    beginInputTimestamp
    endInputTimestamp
    abstract {
      id
      value
    }
  }
}

Server processes through layers:

Resolver (resolver.ts)
  → validates input
  → extracts context (user, observer, ogm)
    │
Service (service.ts)
  → business logic
  → creates/finds Abstract
  → links Manifestation to Session
    │
Neo4j OGM
  → translates to Cypher queries
  → persists to Neo4j
    │
Returns typed output

Survey Data Flow

Surveys use a hybrid architecture combining SurveyJS for UI with Neo4j for storage:

1. SurveyTypeVersion generates a SurveyJS model for UI rendering
2. User interacts with the SurveyJS interface
3. Survey entity stores serialized SurveyJS state for pause/resume
4. On completion, individual Response entities are created in the graph
5. Each Response links to its corresponding Question via a TO relationship

Observer completes Survey
    │
    ▼
For each Question in SurveyTypeVersion
    │
    ▼
Observer provides Response
    │
    ▼
Response links to specific Question
    │
    ▼
Question validates against QuestionScale (if applicable)

Sequence Orchestration

Sequences combine multiple sessions and surveys into ordered protocols:

1. Sequence implements a SequenceTypeVersion
2. SequenceTypeVersion defines ordered SequenceSteps
3. Each step references either a SessionTypeVersion or SurveyTypeVersion
4. As the observer progresses, Sessions and Surveys are created and linked
5. FOLLOWED_BY relationships maintain temporal ordering between steps

Synchronous vs Asynchronous

Synchronous Workflows

• User authentication and authorization
• Session creation and CRUD operations
• Real-time experience reporting during active sessions
• Survey completion and response creation

Asynchronous Workflows

• Background data processing and analysis
• Email notifications and external API calls
• Data export and import operations
• Migration scripts and maintenance tasks

Type Generation Pipeline

When the GraphQL schema changes, types must be regenerated:

TypeGraphQL Models/Inputs/Outputs
    │
    ▼  yarn build:types:all
    │
Server Generated Types
    │
    ▼  Shared via workspace
    │
Client TypeScript Types
    │
    ▼
Apollo Client typed queries

This pipeline ensures end-to-end type safety from the database schema through to the React component props.

Caching Strategy

Layer	Technology	What's Cached
Client	Apollo Client Cache	GraphQL query results, optimistic updates
Client	SQLite / IndexedDB	Offline data, draft sessions
Server	Redis	Session tokens, temporary state
Server	BullMQ (Redis)	Background job queue state