Make sure we exclude most patches when building for upstream
    
    For upstream builds, we want to exclude backports from upstream and
    backwards compatibility patches to make sure backwards compat issues
    are surfaced in test environments so they can be fixed by those
    affected.