Jetpack “Speed up image” messes gallery code

The WP gallery code is fairly streamlined:

    <div id='gallery-1' class='gallery galleryid-2549 gallery-columns-3 gallery-size-full'>
        <figure class='gallery-item'>
            <div class='gallery-icon landscape'>
                <img width="1200" height="1200" src="https://www.../>
            </div>
        </figure>
        <figure class='gallery-item'>
            <div class='gallery-icon landscape'>
                <img width="1024" height="1024" src="https://www..."/>
            </div>
        </figure>
        <figure class='gallery-item'>
            <div class='gallery-icon landscape'>
                <img width="1024" height="1024" src="https://www..."/>
            </div>
        </figure>
    </div>

If I go to JetPack > Performance > Enable site accelerator > Speed up image load times, the gallery code becomes:

<div class="tiled-gallery type-rectangular tiled-gallery-unresized" data-original-width="640" data-carousel-extra='null' itemscope itemtype="http://schema.org/ImageGallery">
        <div class="gallery-row" style="width: 640px; height: 427px;" data-original-width="640" data-original-height="427">
            <div class="gallery-group images-1" style="width: 427px; height: 427px;" data-original-width="427" data-original-height="427">
                <div class="tiled-gallery-item tiled-gallery-item-large" itemprop="associatedMedia" itemscope itemtype="http://schema.org/ImageObject">
                    <meta itemprop="width" content="423">
                    <meta itemprop="height" content="423">
                    <img data-attachment-id="8360" data-orig-file="https://www..."/>
                </div>
            </div>
            <!-- close group -->
            <div class="gallery-group images-2" style="width: 213px; height: 427px;" data-original-width="213" data-original-height="427">
                <div class="tiled-gallery-item tiled-gallery-item-small" itemprop="associatedMedia" itemscope itemtype="http://schema.org/ImageObject">
                    <meta itemprop="width" content="209">
                    <meta itemprop="height" content="210">
                    <img data-attachment-id="8360" data-orig-file="https://www..."/>
                </div>
                <div class="tiled-gallery-item tiled-gallery-item-small" itemprop="associatedMedia" itemscope itemtype="http://schema.org/ImageObject">
                    <meta itemprop="width" content="209">
                    <meta itemprop="height" content="209">
                    <img data-attachment-id="8360" data-orig-file="https://www..."/>
                </div>
            </div>
            <!-- close group -->
        </div>
        <!-- close row -->
    </div>

I truncated some of the code since this is enough to show the problem. Groups and rows are being added to my gallery.

How can I have Jetpack optimize the images only but not mess with the gallery itself?