This is a bug I found on 11.0.3. I tested in TIFF and PSD files, same bug appears.
1. First scenario, works as expected:
I have "LandscapePhoto.NEF" (width is bigger than height, meaning horizontal photo). I edit it on Capture One, I add layers, etc. I don't crop it. I select Edit In -> Photoshop, I then get a "LanscapePhoto.tif". I can copy ALL the settings from my NEF file, including layers, local masks, local adjustments, everything. This is a regular workflow if I want to save a handmade mask and reuse it later in the process. Works.
2. Second scenario, doesn't work as expected:
I have "PortraitPhoto.NEF" (height is bigger than width, meaning vertical photo). I edit it on Capture One, I add layers, etc. I don't crop it. I select Edit In -> Photoshop, I then get a PortraitPhoto.tif". I can copy all the settings from my NEF file, EXCEPT layers, local masks, local adjustments. A dialog saying "Cannot apply local mask" appears. This breaks my workflow since I can't copy settings from layers straight away and I can't reuse my layer masks.
3. Third Scenario, Proof that is a bug:
I have the original "PortraitPhoto.NEF" again, I rotate it 90°clockwise (which makes it a horizontal photo), I select Edit In -> Photoshop, I then get a "PortraitPhoto.tif" but the image is actually horizontal. I can copy ALL the settings from my NEF file, including layers, local masks, local adjustments, everything.
4. Fourth Scenario, Proof that is a bug v2:
I have the original "PortraitPhoto.NEF" again, I select Edit In -> Photoshop, I then get a "PortraitPhoto.tif". I can copy all the settings from my NEF file, EXCEPT layers, local masks, local adjustments. A dialog saying "Cannot apply local mask" appears.
I rotate the NEF 90°clockwise on Capture One, and I rotate 90°clockwise and save the TIFF on Photoshop. Now I can copy everything including layers and mask. If I rotate the TIFF back again in Photoshop, the file updates to a vertical file and I get the error again.
5. Fifth scenario, Proof that is a bug v3 and somewhat of a WORKAROUND for anyone else having this bug and wanting to copy layers and masks on vertical/portrait files:
I have the original "PortraitPhoto.NEF" again, I select Edit In -> Photoshop, I then get a "PortraitPhoto.tif". I can copy all the settings from my NEF file, EXCEPT layers, local masks, local adjustments. A dialog saying "Cannot apply local mask" appears.
I rotate the NEF 90°clockwise on Capture One, and I rotate 90°clockwise and save the TIFF on Photoshop. Now I can copy everything including layers and mask. I rotate the TIFF file back 90°counterclockwise on Capture One leaving me with a vertical file with the layers copied. The strangest part of this, and the definitive proof that is a bug is that in this scenario my NEF is vertical and my TIFF is horizontal at the time of copying the mask!
^This workaround is nowhere near ideal, since I'm doing a destructive transformation of the file on Photoshop, it could be a final step on the edition workflow and it wouldn't be a problem, but if I want to come back and edit the TIFF later, I have to rotate it again to work on it and rotate it back so it's horizontal, also, each time I save it vertical, Capture One will erase the layers and masks, and the settings I made, so it's just, not helpful at all.
Right now the closest ideal workflow is working on the horizontal file in Photoshop, rotating the view, so it looks vertical while you work. But in the finder and elsewhere all the TIFF files are going to be horizontal, which is bad.
My point is, I should be able to copy layers and local mask regardless of the file orientation.